We created 20 files with random data, imitating taxon by locality matrices, through a program written in Visual Basic. The random numbers were created by means of the random generator of Visual Basic, using the decimal fractions only and placing one digit in each cell. In the resulting matrices of 20 rows by 10 columns the numbers 0 to 9 occur in more or less equal frequencies, and no structure whatsoever is expected to exist in such a matrix. These 20 matrices are represented in Table 1 on the line “standard”.

Inside each matrix the contributions of the individual cells to the matrix sum vary only moderately, so we created a derivate of each matrix by multiplying certain values by 10, in order to get a distribution of several high values and many low values per row and column, a situation that is probably more similar to a real taxon by locality array. These 20 matrices are represented in Table 1 on the line “adapt”.

A second set of derivate matrices was created by substituting randomly the contents of about 35% of the cells by 0. These 20 matrices are represented in Table 1 on the line “zero”. The line “pattern” indicates matrices with added zeros, created by an algorithm that introduces some kind of a pattern in the data. For details see the chapter “The influence of zeros in a matrix”.

Each matrix was submitted to CA using PAST version 1.71 [8]. Per matrix group (standard, adapted, zero and pattern) the minimum, mean and maximum percentages were calculated for the first axis, the sum of the first two axes, and the sum of the first four axes (Table 1).

Then, the entire process was repeated for matrices of 30 rows by 15 columns, matrices of 40 rows by 30 columns and matrices of 98 rows by 30 columns, resulting in a total of 320 matrices.

The results were first calculated over five and 10 matrices in each group; between the results for five and 10 matrices there were some important differences; the results for 10 and 20 matrices are very similar, so we may assume a sample of 20 matrices is sufficiently reliable. In a few cases we analyzed up to 60 equally sized matrices and this confirmed that 20 is a sufficiently large number.

This procedure allows an analysis of the correlation between matrix size and axis percentages in CA. However, another factor in CA is the inertia, or sum of the eigenvalues. In our random files the total inertia rarely exceeds 1.0, which is considered to be a low value, caused by the fact that the majority of the values range between 0 and 9, and higher values are scarce in the adapted matrices, and absent in the other two groups.

Therefore, we created new random matrices, in which we randomly multiplied certain cells by stepwise increasing factors, incrementing the total inertia to values over 6.0. We analyzed the correlation of increasing inertia with decreasing values of the first axis of CA.

Finally, we analyzed a set of matrices with 70% zeros, as frequently found in palaeontological practice.

Since the matrices contain random data that, in principle, present no correspondence (except maybe for some fortuitous case), one has to admit that conclusions based on real cases with similar axis values are invalid. Valid conclusions based on the axis percentages of CA can only be drawn on values that are considerably larger than the results obtained from the random matrices.

The size of the array is strongly related with the percentages obtained on the first axes (Fig. 1). The percentages of the first four axes in a large matrix are considerably smaller than in a 20 × 10 matrix. In real cases, the threshold from where results may be considered to be useful should be placed much higher in small arrays. Apart from that, one should consider whether —independent of the array size— results of less than about 70% for the first four axes are useful.

The values obtained for the “adapted” matrices are constantly lower than those for the corresponding “standard” matrices. Assuming that the method of adaptation used did not introduce structure into the matrix (and there is no reason to believe it did), one must conclude that a relatively small number of cells with much higher values than the majority has a considerable influence on the results obtained. Ter Braak [3] suggested logarithmic transformation for such matrices. We tried this, and in some cases the result for the first axis in the log-transformed matrices was lower than in the original matrix, but in quite some cases considerably higher values were found. Apparently, logarithmic transformation produces unpredictable results.

Plotting the first two axes of CA may give diffuse distributions, or there are some groups and isolated points, for both the standard and the adapted matrices. In the largest matrices no groups or outlying points can be recognized.

On Fig. 1 the mean values and the maxima of the first four axes of the standard matrices are plotted against the number of cells. The correlation between the number of cells and the percentages of the axes is evident. The letters c, g, k, m and p on Fig. 1 represent the position of real data taken from the literature that will be discussed afterwards.

For the decision whether the results of CA for a real matrix are meaningful, we have to take as a threshold the maximum value found in our random data, for the corresponding matrix size, plus a certain margin. In practice this means that the results of a medium-sized matrix (500–1000 cells) should be considered insufficient when the value of the first axis is below 25–30%, or the sum of the first two axes is below 35–45%. In large matrices the threshold should be chosen at 20% (one axis) and 30% (two axes). For small matrices the thresholds are around 40 and 60%, and these percentages must be considered conservative estimates.

An additional problem with big matrices with low axis values is the number of axes to be interpreted. Interpretation of more than four axes is practically impossible. So, probably no valid conclusions can be drawn when the first four axes cover less than 70% of the inertia, independent of the size of the matrix.

On the other hand, this does not mean that there is no correlation in a matrix with low axis values. It only means that in such cases no valid conclusions can be based on CA, and visual inspection of the matrix may be more fruitful.

We took the number of cells as a measure for the matrix size, but the relation between the number of columns and rows has some influence too: as a general rule square tables give lower axis values than oblong tables; e.g. in ten 50 × 60 standard matrices the value of the first four axes are constantly lower than in the 98 × 30 tables, which have practically the same number of cells. In the same way the values for 14 × 14 tables are slightly lower than for 20 × 10 tables (Table 1). When one decreases the value of one of the dimensions of the table, maintaining more or less the same number of cells, the values of the first four axes of CA will increase until —of course— reaching 100% in matrices with only five columns or five rows. In such matrices one should analyze only one or two axes, and these should give very high values in order to be meaningful. In small matrices at least one of the dimensions is necessarily small and that is one of the reasons why they show higher axis values than large matrices.

When discussing the results with Dr Casanovas-Vilar (Sabadell) the question arose whether the total inertia could influence the results. In our standard and zero matrices the values of the cells vary between 0 and 9; in real matrices the differences between cells are usually much greater, resulting in a greater inertia. We therefore refined the method of multiplication applied to the adapted matrices, and found that there is some correlation between total inertia and the percentages found for the first axis, depending on the size of the matrix (Fig. 2).

For each of the classes of 1200, 450 and 200 cells we created matrices in which we increased the inertia through six steps of multiplications of cell values by increasing factors, returning to the original matrix after each step. We did this in two different ways. In the first algorithm we multiplied the same cells in these six steps. This does not change the structure of the matrix, it merely stretches the range of the values. In the second algorithm, in each step we randomly chose the cells to be multiplied, creating six matrices with different structures. There were no important differences in the results of these two methods.

For the resulting 20 × 10 matrices the variability is very great and the points are distributed in an irregular way, but higher values on the first axes seem to be correlated with lower eigenvalues, though in some matrices the opposite is the case. On Fig. 2 the consecutive multiplication steps for each 20 × 10 matrix are connected, so one can see that in several cases the first step causes a strong increase of the percentage of the first axis; after that there is a negative correlation.

For large matrices there is practically no correlation, the points plot on an almost vertical line.

Creating random matrices with a high percentage of zeros is not easy, and several algorithms were rejected because they apparently introduced rhythmic sequences in the matrix, often recognized by the fact that the first two axes gave practically the same values. When we analyzed graphic representations of such matrices (coloring the background of zero cells [Fig. 3]), we observed in quite some cases diagonal or V-shaped patterns of zeros. These matrices gave very high percentages for the first axes, in comparison with the standard matrices they were derived from (see line “pattern” in Table 1), but in fact they are no longer random matrices. Before executing CA on a real matrix one should analyze it, to make sure that there is no meaningless accidental pattern that would influence the results of CA.

Ter Braak ([3], table 5.3) noted the influence of diagonal structures on CA. A hidden diagonal structure may become visible by reordering the rows (and maybe that is what CA does, because randomly reordering the rows has no influence on the axis values); however, in a real matrix where the rows are placed in stratigraphic order, such reordering would be senseless because it approaches information that by nature is separated.

The rejected algorithm revealed a second problem: the lines “pattern” in Table 1 refer to matrices with 35% zeros that form a pattern. The values for the CA axes are much higher than in the matrices they were derived from. We repeated the same procedure substituting with “1” and “2”. In these cases the axis values do not deviate significantly from those of the standard matrices. Apparently a pattern formed by zeros has a much greater influence on CA results than a pattern formed by a nonzero value.

The matrices produced by the correct algorithm, not introducing a pattern, are represented on the lines “zero” in Table 1. They are not significantly different from the matrices they were derived from.

The matrices of Casanovas-Vilar and Agusti [4], García-Alix et al. [5] and Minwer-Barakat [10] have about 70% of zeros. Line A in Table 2 lists the values obtained for 20 random 35 × 30 matrices with 70% zeros. Their inertia varies between 2.8 and 3.5, mean 3.1. Apparently the high proportion of zeros produces a very great variability, and some very high values for the first axis. We repeated this for matrices, where randomly chosen cells were multiplied to obtain greater total differences between cells. Their inertia varies between 3.6 and 4.6, mean 3.9, but the maxima obtained for the axes do not differ substantially from the previous case. The results are given in Table 2, line B.

Sometimes CA is applied to absence/presence matrices that contain only zeros and ones. We transformed five random matrices of each size group to such absence/presence matrices, and found nearly always an increase of the value of the first axis of CA. In the 20 × 10 matrices the greatest increase found was from 24.0 to 41.3%, and since we tried only five matrices greater increases are certainly possible. In the 30 × 15 arrays the increase was about 5%, with one exception: from 17.9 to 27.2%. In the larger matrices only slight increases were found. In the 30 × 35 matrices with 70% zeros the maximum increase found was from 14.0 to 21.5%. Moreover, whereas the original matrices normally give a diffuse plot, these random absence/presence matrices tend to show clear groupings in the plot of the first two axes.

When applying CA to absence/presence matrices, one must consider a higher threshold to decide whether the results are meaningful.

Apart from CA we submitted our 20 × 10 and 98 × 30 matrices to PCA. The results, represented in Table 3, are quite similar to the results of CA. This may mean that the same considerations presented here for CA apply to PCA too.

We compared the results of our random data with real matrices taken from Casanovas-Vilar and Agustí [4], García-Alix et al. [5], Koufos [9], Minwer-Barakat [10] and Popov [11].

On Fig. 1, c1, c2 and c4 represent the first axis, the sum of the first two axes and the sum of the first four axes of the Casanovas matrix (1015 cells). C1 is very close to the range of random matrices, c2 and c4 are outside the range of random matrices.

The c on Fig. 2 also represents the Casanovas matrix. The value of the first axis: 18.1, with a sum of the eigenvalues of 4.48 (pers. comm. Dr Casanovas-Vilar), scores better than our random matrices. By interpolation one may estimate the maximum of the first axis value for a 1000 cell random matrix to be between 11 and 12 (in a few trials we found a maximum of 11.0). It is difficult to say whether the difference between 12 and 18 is large enough to conclude that the Casanovas matrix is outside the zone of random matrices. On the other hand, the sum of the first four axes: 57.72, is so low that the interpretation may easily be incorrect.

The Casanovas matrix contains about 70% zeros, so we created 20 random matrices of the same size (30 × 35) with 70% zeros and 20 matrices with 70% zeros and multiplication of values (Table 2), using the algorithm that does not introduce a pattern. In both cases the sum of the eigenvalues is comparable to the value found by Casanovas-Vilar and Agusti [4], and the maximum values obtained for the axes come so close to the values in the Casanovas matrix that one must conclude that the latter are not significantly different from a random result.

On Fig. 1, g1, g2 and g4 represent the axis data of the García matrix [5]. The values obtained are well above the limits calculated from the random files, but we could not make a useful interpretation of these results, and think that in this case they are fortuitous. The same goes for m1, m2 and m4 of the Minwer matrix [10]. On Fig. 2 both these matrices fall outside the range of random matrices in view of the number of cells they contain.

On Fig. 1, k1, k2 and k4 are the data for the Koufos matrix [9]. The values are very high, and their interpretation by Koufos appears to have a sound basis. This is confirmed on Fig. 2, in spite of the low value of the inertia.

The Popov matrix (p on Fig. 1) is a 34 × 7 absence/presence matrix (238 cells). On Fig. 1 and Fig. 2 it falls within the range of the standard random matrices. As said before, we found a value of 41% for the first axis of CA in a 20 × 10 absence/presence matrix, which is considerably larger than the 30.1% found by Popov [11]. The conclusions of Popov may be perfectly correct, but they cannot be inferred from the results of CA.

Conclusions obtained from CA cannot be evaluated correctly when the total inertia and the matrix size are not given. In publications of the results of CA this information should be available. Another indispensable datum is the percentage of zero cells.

CA is doubtlessly a useful technique. But, the high values obtained from random matrices demonstrate that one should be careful when using CA for analyzing real data matrices; the obtained values should be well above the threshold values presented here. This is especially true when these matrices are not very big, or when they contain a high percentage of zeros. Simple visual inspection of a data matrix is then probably more reliable.

For the decision whether the numerical results of CA for a real matrix are meaningful, we have to consider a threshold based on the results from the random matrices. For small matrices (up to 500 cells) the first axis should represent at least 40% of the inertia, and the first two axes should sum 60%. In a medium-sized matrix (500–1000 cells) these limits are 30 and 40%, respectively. In large matrices the thresholds should be chosen at 20% (one axis) and 30% (two axes). In such cases, however, serious problems arise, because one has to interpret too many axes, and, probably, one should refrain from using CA when the sum of the first four axes is less then 70%. For matrices with many zeros and for absence/presence matrices the thresholds are higher than stated before.

It is necessary to check whether a matrix contains an accidental hidden diagonal structure, which may result in a high but meaningless value for the first axis.